The difference is minuscule in the long scheme of exercise and burning calories.
Do what helps you get through the swim workout. If you normally eat first thing in the morning, don't go against your nature. If you can't exercise while on a full stomach, then don't eat beforehand.
If you normally eat in the morning, BUT you can't exercise on a full stomach, just eat a little bit, like an apple or a piece of toast or something.